Method and apparatus for enabling redundancy in a network element architecture

ABSTRACT

A network element includes a plurality of Input/Output Cards (IOCs), a plurality of Datapath Service Cards (DSCs); and at least one crosspoint switch card (XPC) configured to be able to selectively interconnect each of the IOCs with each of the DSCs. Enabling full interconnectivity between all the IOCs and DSCs enables greater sparing options within the network element. Additionally, network element is configured to enable the XPCs to be spared, thus eliminating the XPCs as a potential single source of failure in the network element.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to Provisional U.S. Patent ApplicationNo. 60/561,379, filed Apr. 12, 2004, and also claims priority toProvisional U.S. Patent Application No. 60/569,717, filed May 10, 2004,the content of each of which is hereby incorporated herein by reference.

BACKGROUND

1. Field

The present invention relates to communication networks and, moreparticularly, to a method and apparatus for enabling redundancy in anetwork element architecture.

2. Description of the Related Art

Data communication networks generally include numerous routers andswitches coupled together and configured to pass data to one another.These devices will be referred to herein as “network elements.” Data iscommunicated through the data communication network by passing protocoldata units, such as packets, frames, cells, or segments, between thenetwork elements over one or more communication links formed usingoptical fibers, copper or other metallic wires, or wireless signals. Aparticular packet may be handled by multiple network elements and crossmultiple communication links as it travels between its source and itsdestination over the network.

FIG. 1 illustrates an example communication network in which local areanetworks 10 are connected by a network element 12 to another networkdomain 14 having high speed communication links 16 interconnecting highspeed network elements 18. The several network elements may beimplemented in a similar fashion or differently depending on theirintended use on the network. While FIG. 1 is a relatively simplisticrendering of a communication network, in reality the networks can getquite complicated. As shown in FIG. 2, multiple network elements aregenerally housed together in a communication center and may be mountedin a racking system 19 or, alternatively, may be free-standing.

As communications networks increase in size and sophistication, thedemands put on the network elements have likewise increased to the pointwhere a given network element may be configured to handle multipleterabits of data per second. To accommodate this increased amount ofdata traffic, the internal architecture used to implement networkelements has changed over time. Generally, a network element has acontrol plane and a data plane. The control plane controls overalloperation of the network element and the data plane is configured oroptimized to handle data traffic on behalf of the network element. Forexample, a typical data plane includes circuitry configured to interfacethe communication links, such as to receive the physical signals fromthe communication links, extract data from the physical signals, performnoise reduction and other signal processing functions, and optionallygroup the received signals into packets or other logical associations ofbits and bytes. This type of initial processing will be called I/Oprocessing.

In addition, a typical data plane will process packets or frames of datato cause those packets/frames to be switched or forwarded onto one ormore communication links. This additional processing will be referred toas datapath service processing, which may include extracting informationfrom a header or label associated with the packet, or other functionsthat may be necessary or desirable to be performed in connection withthe packet/frame or stream of packets/frames.

Previous generation network elements, and some existing networkelements, used functional cards in which both I/O processing functionsand datapath service functions were performed on the same network card.A network card, or “functional card” as that term is used herein, isgenerally formed from a printed circuit board on which processingcircuitry is implemented. The functional cards are plugged into aconnector plane in much the same way as a memory card is plugged into amotherboard of a computer. FIG. 3 illustrates a set of functional cards20 plugged into a connector plane at the rear of a network element(referred to herein as a backplane 22) and FIG. 4 illustrates a set offunctional cards plugged into a connector plane in the middle of thenetwork element (referred to herein as mid-plane 24).

Using integrated functional cards began to exhibit drawbacks as networkdata rates increased and advances were made in the processing circuitryart. For example, a given functional card can only host a limited numberof connections to physical communication links, such as fiber optic andwire connections, due to the finite amount of space available on thefront edge of the card to carry the line connectors. As processingtechnology increased, datapath services processing circuitry on theboard advanced so that it was able to handle more traffic than could beconnected to the functional card over this limited number of connectors.Thus, using integrated functional cards required the datapath servicesaspect of the network element to be overbuilt.

To solve this problem, an architecture was developed in which differentfunctional cards were used to implement I/O and datapath servicefunctions. A functional card that is configured to perform Input andOutput functions will be referred to herein as an IOC, and a functionalcard that is configured to perform datapath service functions will bereferred to herein as a DSC. This architecture has been widely adoptedand many large network elements now include a plurality of IOCs, aplurality of DSCs, and optionally one or more other functional cardssuch as server cards, all interconnected by a midplane or backplane.Mid-planes and back-planes will collectively be referred to herein as“connector planes.” In a mid-plane architecture, the IOCs are generallyinserted into connectors on the front of the mid-plane from the front ofthe network element, and DSCs and other processing cards are insertedinto connectors on the back of the mid-plane from the rear of thenetwork element. In addition, as shown in FIG. 5, the functional cardsmay be full height cards such as cards 26 or fractional height cardssuch as cards 28.

In addition to increasing the network element's ability to handleincreased amounts of data, it was also desirable to increase thereliability of the network element. One way to do this is to provideredundant functional cards such that if one of the IOCs or DSCs fails,another IOC or DSC can be automatically used in its place until thefailing IOC or DSC is able to be replaced. The extra IOC or DSC will bereferred to herein as a “spare” functional card. A functional card thathas a dedicated spare will be referred to as being spared in a “1:1”fashion while a group of functional cards that are spared by a singlefunctional card will be referred to as spared in a “1:n” fashion.

In conventional network elements, the paths through the midplane orbackplane were fixed such that particular IOCs were required to connectto a particular DSC and vise versa. This result was dictated by thedesign of the traces on the connector plane. The result of thisarchitecture was that the number and type of sparing that could beperformed, and the manner in which interconnection of the functionalcards could be implemented, was limited by the midplane or backplanedesign.

One way to avoid limitations of this nature is to provide full meshinterconnectivity between all functional cards in the network elementvia the connector plane. However, such a solution does not scale well.Specifically, if the connector plane is required to have a traceconnecting every functional card to every other functional card, thenumber of traces on the midplane will be on the order of n², where n isthe number of functional cards. Thus, while this solution may work wherethere is a limited number of functional cards, it becomes very difficultand ultimately cost prohibitive to create a midplane or backplane thatis able to provide full mesh interconnectivity between IOCs and DSCs asthe number of functional cards in use on the network element increases.

One attempt to provide greater interconnectivity without implementing afully meshed midplane architecture is described in U.S. ProvisionalPatent Application No. 60/402,761, to Bradbury, et al., entitled“Redundancy Crossbar Card”, in which two crosspoint switches are used tointerconnect groups of line cards and accelerator cards. In the Bradburyarchitecture, the line cards are separated into two groups with half ofthe line cards connected to one crosspoint switch and the other halfconnected to the other crosspoint switch. The accelerator cards areconnected to both crosspoint switches.

While the Bradbury architecture allows sparing of accelerator cards in a1:1 or 1:n manner, it does not allow the line cards to be spared in asimilar manner. Specifically, since each line card is connected to onlyone crosspoint switch, failure of that crosspoint switch will cause afailure of all associated line cards. To prevent a failure of thisnature from affecting traffic passing through the network element, theprotection cards in Bradbury's architecture are required to be sparedvia a line card attached to the other crosspoint switch. Specifically,to avoid the crosspoint switches from becoming a single point of failurein the network element, line cards from one group of line cards arerequired to be spared by line cards in the other group. This places arestriction on the manner in which sparing may be implemented and alsolimits the number of cards that may be active on the network elementsince half of the line cards must be reserved as spare line cards.Accordingly, it would be desirable to provide a network elementarchitecture that is able to enable greater interconnection between thefunctional cards of a network element.

SUMMARY OF THE DISCLOSURE

The present invention overcomes these and other drawbacks by providing amethod and apparatus for enabling full redundancy at the functional cardlevel in a network element architecture. According to an embodiment ofthe invention, a network element, includes a plurality of Input/OutputCards (IOCs), a plurality of Datapath Service Cards (DSCs); and at leastone crosspoint switch card (XPC) configured to be able to selectivelyinterconnect each of the IOCs with each of the DSCs. Enabling fullinterconnectivity between all the IOCs and DSCs enables greater sparingoptions within the network element. Additionally, according to anotherembodiment of the invention, at least one additional XPC may beprovided, and network element is configured to enable the XPCs to bespared as well as the IOCs and DSCs.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present invention are pointed out with particularity inthe appended claims. The present invention is illustrated by way ofexample in the following drawings in which like references indicatesimilar elements. The following drawings disclose various embodiments ofthe present invention for purposes of illustration only and are notintended to limit the scope of the invention. For purposes of clarity,not every component may be labeled in every figure. In the figures:

FIG. 1 is a functional block diagram of an example of a communicationnetwork including network elements;

FIG. 2 is a front view of a plurality of network elements housedtogether in a rack;

FIG. 3 is a perspective view of functional cards connected to abackplane;

FIG. 4 is a perspective view of functional cards connected to amid-plane;

FIG. 5 is a perspective view of functional cards of different heightsconnected to a mid-plane;

FIG. 6 is a functional block diagram of an example selection offunctional cards connected to a mid-plane according to an embodiment ofthe invention;

FIG. 7 is a functional block diagram illustrating signal paths on amid-plane that may be used to interconnect functional cards with one ormore crosspoint switches according to an embodiment of the invention;

FIG. 8 is a functional block diagram of a channel between an IOC and aDSC according to an embodiment of the invention;

FIG. 9 is a functional block diagram of a crosspoint switch according toan embodiment of the invention;

FIG. 10 is a functional block diagram illustrating interconnection ofIOCs, XPCs, and DSCs in a network element, according to an embodiment ofthe invention;

FIG. 11 is a functional block diagram illustrating interconnection ofInput/Output Cards (IOCs) with Datapath Service Cards (DSCs) viaredundant crosspoint switches;

FIG. 12 is a functional block diagram of a network element configured toimplement full redundancy in the data plane according to an embodimentof the invention;

FIG. 13 is a functional block diagram illustrating exampleinterconnections that may be made using the redundant crosspoint switcharchitecture according to an embodiment of the invention;

FIGS. 14 a-14 c are functional block diagrams illustrating 1:1, 1:n, andm:n sparing of IOCs in a dataplane of a network element according to anembodiment of the invention;

FIG. 15, is a functional block diagram illustrating sparing of crosspoint switch cards and IOCs in a dataplane of a network elementaccording to an embodiment of the invention;

FIG. 16 is a functional block diagram illustrating interconnection offunctional cards to enable sparing of IOCs, DSCs, and XPCs, in a dataplane of a network element according to an embodiment of the invention;and

FIG. 17 is a functional block diagram illustrating possible sparingcombinations between IOCs and DSCs in a network element according to anembodiment of the invention.

DETAILED DESCRIPTION

The following detailed description sets forth numerous specific detailsto provide a thorough understanding of the invention. However, thoseskilled in the art will appreciate that the invention may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, protocols, algorithms, and circuits have notbeen described in detail so as not to obscure the invention.

As described in greater detail below, the method and apparatus of thepresent invention enables full redundancy at the functional card levelin a network element architecture. According to an embodiment of theinvention, a network element, includes a plurality of Input/Output Cards(IOCs), a plurality of Datapath Service Cards (DSCs); and at least onecrosspoint switch card (XPC) configured to be able to selectivelyinterconnect each of the IOCs with each of the DSCs. Enabling fullinterconnectivity between all the IOCs and DSCs enables greater sparingoptions within the network element. Additionally, the network elementmay be provided with more than one XPC and configured to enable the XPCsto be spared as well.

FIG. 6 illustrates a number of functional cards connected to a mid-planeaccording to an embodiment of the invention. As shown in FIG. 6,Input/Output Cards (IOCs) 30 and Datapath Service Cards (DSCs) 32 areinterconnected using a connector plane and one or more crosspointswitches (XPCs) 34 such that any IOC may be connected to any other IOC,any IOC may be connected to any DSC, and any DSC may be connected to anyother DSC via any XPC in the network element. Additionally,implementation of an architecture of this nature, as discussed ingreater detail below, enables 1:1, 1:n, and m:n sparing of IOCs, servercards, and DSCs in the network element. M:n sparing will be used hereinto refer to a situation where two or more functional cards may be usedas spares for a group of other functional cards. Additionally, asillustrated in FIG. 6, providing redundant XPCs with fullinterconnection of the IOCs and DSCs to each of the XPCs enables 1:1 or1:n sparing of the XPCs to enable redundancy to be provided to thecrosspoint switches as well as the other functional cards in the networkelement.

In the embodiment illustrated in FIG. 6, a mid-plane has been used toconnect the functional cards together. The invention is not limited inthis manner as a back-plane could be used as well. The invention is alsonot limited to an embodiment using the illustrated number of functionalcards as numerous types and quantities of functional cards may be usedin a network element.

FIG. 7 illustrates interconnections within an example mid-plane toenable redundant XPCs to be used in connection with redundant IOCs andDSCs. As shown in FIG. 7, each IOC is connected to a first XPC via afirst trace (solid line) and is connected to a second XPC via a secondtrace (dashed line). The XPCs are interconnected via a plurality oftraces represented by the heavier weighted solid line 35 which may beused facilitate interconnection of IOCs and DSCs as discussed in greaterdetail below. The XPCs are configured to switch signals from the IOCcard to another IOC or to a DSC via a trace extending from that XPC tothe other functional card. For example, assume that signals received atIOC 1 were to be transferred to DSC 3. Signals from IOC 1 are thustransmitted from IOC 1 to one of the XPCs via traces in the connectorplane, switched by the XPC, and output via the traces on the connectorplane to the appropriate DSC.

By using an XPC to transfer signals between traces on the connectorplane, full interconnection of all functional cards may be achievedwithout requiring the midplane to have n2 traces between each of thefunctional cards. Rather, full mesh interconnectivity may be achievedusing only the order of n traces on the connector plane, and causing thesignals to traverse the midplane two times—once from the IOC to the XPC,and once from the XPC to the DSC.

As shown in FIG. 7, the functional cards such as the IOCs and DSCs mayhave multiple connections to the mid-plane. For example, in theillustrated embodiment the IOCs each have three pairs of unidirectionallinks (one of each pair of links carrying data from the IOC to themidplane and the other of each pair of links carrying data from themidplane to the IOC), with each link capable of carrying data at up to3.125 or other convenient line rate. The invention is not limited to theparticular link speeds used to implement an embodiment of the invention.Similarly, each of the DSCs is connected to the connector plane using 12pairs of unidirectional links operating at similar bandwidths.Connecting the DSCs to the midplane using 12 pairs of links allows agreater number of IOCs to connect to each of the DSCs to thereby reducethe number of DSCs required in the network element.

In the embodiment illustrated in FIG. 7, the IOCs may connect to morethan one DSC by causing the XPC to direct signals on the three links todifferent functional cards. Thus, one set of signals being transferredon a first of the three links connecting the IOC to the connector planemay be handled by a first DSC, a second set of signals on the second ofthe three links may be handled by a second DSC, and a third set ofsignals on the third of the three links may be handled by a third DSC.This allows for greater flexibility in the type of communications thatmay be handled by a given IOC and, hence, by the network element. Forexample, Packet Over SONET (POS) traffic may be sent to one DSC, ATMtraffic may be sent to another DSC, and a third type of traffic may besent to another DSC. While an embodiment in which three pairs of linksare used to connect the IOC to the connector plane has been describedherein, the invention is not limited in this manner as any number oflinks may be used to connect the IOCs to the connector plane. Similarly,the invention is not limited to a DSC that uses twelve pairs of links toconnect to the connector plane as other numbers of links may be used toconnect these components of the network element.

In the illustrated embodiment, since each IOC has 3 links to theconnector plane and each DSC has 12 links to the connector plane, eachDSC can handle the traffic from between one and twelve IOCs. To increasethis number, the number of links from the DSC to the connector plane maybe increased or traffic from multiple IOCs may be aggregated at anotherIOC before being transmitted to the DSC. For example, traffic fromseveral IOCs may be sent to another IOC, aggregated, and then forwardedfrom that IOC to the DSC using a single link. Using a single level ofcompression of this nature allows 36 IOCs to connect to one DSC,provided that the IOCs and DSCs are connected as discussed above, andfurther provided that there are 36 IOCs in the network element to beconnected to the designated DSC.

One advantage of allowing multiple IOCs to be connected to any given DSCis that Automatic Protection Switching (APS) in a SONET network may beperformed at the DSC rather than at the IOC which reduces the need tohave the IOCs handling the traffic for a particular SONET ring to beconnected together.

To understand this advantage, it may be useful to look at FIG. 7 andassume, for the sake of this example, that a SONET ring is provisionedthrough IOCs 1 and 2. In a SONET network, a SONET ring has protectionand working fibers extending around the ring and network elements on thering always transmit traffic onto both working an protection paths toaccelerate protection switching between the paths. On the receive side,the SONET traffic is pulled off of the working fiber or protection fiberdepending on the state of the ring. One aspect of this selection isreferred to as Automatic Protection Switching (APS).

Conventionally, to allow IOC sparing in a SONET switch, the IOCs wouldbe physically linked together and APS selection of one of the incomingSONET streams of traffic would be performed by the IOCs. According to anembodiment of the invention, by allowing each of the IOCs to beconnected to any arbitrary DSC, APS switching may instead be performedby the DSCs rather than the IOCs, to allow a given DSC to select one ofthe incoming SONET streams, so that the fibers forming the SONET ringmay be homed to different IOCs without requiring the IOCs to beseparately interconnected.

The network element of FIG. 7 may also include one or more IOC servercards configured to provide services on the network element. Forexample, the server cards may process traffic to be output over thenetwork, such as to encrypt the traffic, replicate the traffic, orotherwise alter the content of the data. For example, the server cardsmay execute security, VPN, and other services for the network element.Other services may be performed as well and the invention is not limitedto an embodiment that implements these particular selection of services.

FIG. 8 illustrates a channel extending between an IOC and a DSC in anetwork element according to an embodiment of the invention. As shown inFIG. 8, the channel includes data traffic including packets, cells,frames, or other protocol data units, and control traffic configured toenable control messages to be passed between the IOC and DSC cards.According to an embodiment of the invention, traffic (such as SONET,Ethernet, and/or TDM traffic) is terminated at the IOC and formed intopackets or frames at a framer on the IOC. The packets or frames are thenpassed over the channel to the DSC for processing and/or switching.Passing data between the IOC and DSC in packet format allows the networkelement to handle traffic on a per-packet basis, rather than on anotheraggregate basis such as a STS-1 basis, to achieve finer granularity ofcontrol over traffic passing through the network. The invention is notlimited to an embodiment that performs functions on a per-packet basis,however, as other manners of handling data traffic may be used as well.

Each XPC 34 contains a statistically configured fully meshed cross pointswitch which provides for point to point interconnections between inputand output ports. Since the input and output ports are connected totraces on the connector plane, the crosspoint switch allows any twotraces on the connector plane to be connected to thereby enable any twofunctional cards to be interconnected. One example of an XPC is shown inFIG. 9, although the invention is not limited to this type of XPC asmany types of XPCs and similarly configured switching architectures maybe developed. Providing point-to-point connections between inputs andoutputs provides faster interconnection than another architecture, suchas a bus, in which only one input may be transmitting at a given timeover the transmission mechanism. Likewise, the statically configuredpoint-to-point crosspoint switch 36 is much less expensive than adynamic switch structure. Thus, an XPC may be used, in one embodiment,to provide initial interconnectivity between IOCs and DSCs. As discussedbelow, the network element may use a switching fabric at a later stageof handling the packets, which may include a non-blocking dynamic switchstructure, to switch signals between ports on IOCs. Use of a switch ofthis nature at a later stage to switch the signals is thus not precludedby use of a crosspoint switch to interconnect IOCs and DSCs for initialpacket processing in the front end of the network element.

As shown in FIG. 9, the XPC card includes one or more crosspointswitches 36 as well as control circuitry 38 configured to enable thecrosspoint switches to be controlled to selectively interconnect inputand output ports. Control of the operation of the XPC generally causesinterconnections to be made by latch mechanisms 40 at junctions betweeninput lines and output lines. In the illustrated embodiment, activatedlatch mechanisms are illustrated as filled squares and inactive latchmechanisms are illustrated as empty squares.

The XPC may be accessed by a control program via interface 42. Althoughthe XPC is described as being static, the connection of inputs andoutput ports can change over time as components fail, to allow sparingto occur on the network element and to allow configuration changes to beimplemented on the network element. Thus, the term static implies aconnection that does not change every time a new data packet is to behandled by the crosspoint switch.

The crosspoint switch, in addition to being fully meshed, isnon-blocking such that signals arriving at an input are not blocked fromarriving at their output destination. Several commercially availablecrosspoint switches that may be used to implement an embodiment of theinvention are made by MindSpeed™, particularly the MindSpeed™ M21151crosspoint switch and MindSpeed™ M21156 switches, although othercrosspoint switches such as the M21131 and M21136 crosspoint switchesmade by MindSpeed™ or another manufacturer may be used as well. Both ofthese identified switches are 144×144 3.2 Gbps cross point switches, onewhich includes clock data recovery using an integrated phase locked loopand the other of which does not. The invention is not limited to anarchitecture using a switch of this size or a switch having theseparticular features, as many different cross point switches may be usedto implement embodiments of the invention.

FIG. 10 illustrates an embodiment of the invention in which one XPChaving two crosspoint switches is provided to enable interconnectivitybetween IOCs and between DSCs to be established. In the example shown inFIG. 10, particular numbers of IOCs, XPCs, and DSCs are illustrated anddescribed. The invention is not limited to this particular example asmany different numbers of these functional cards may be used withoutdeparting from the scope of the invention. Specifically, in the Exampleillustrated in FIG. 10, the network element includes 24 IOCs, one XPChaving two crosspoint switches, and 8 DSCs. Each of the IOCs isconnected to the XPC using three bi-directional links or a total of sixconnections. Each of the DSCs is connected to the XPC using 12bi-directional inks or a total of 24 connections. The crosspointswitches on the XPC are connected to each other to allow IOCs to beconnected to other IOCs without going through a DSC and to allow DSCs tobe connected to DSCs without passing through an IOC. More particularly,in this example, crosspoint switch-1 is configured to use 72 inputconnections to service 3 links from each of the 24 IOCs, and to use 96output connections to service 12 links to each of the 8 DSCs. Crosspointswitch-2 is similarly configured to use 96 input connections to service12 links from each of the 8 DSCs, and to use 72 connections to service 3links to each of the 24 IOCs. The remaining links (72 input links fromcrosspoint switch-2 to crosspoint switch-1 and 48 links from crosspointswitch-1 to crosspoint switch-2) are used to respectively provideDSC-DSC connectivity and to provide IOC to IOC connectivity. In thisillustrated embodiment, a total of 168 input and output lines wererequired to provide full interconnectivity between the IOCs and DSCs.Since the available crosspoint switch had only 144 input and outputlines, two crosspoint switches were used to provide fullinterconnectivity on the one crosspoint card. As larger crosspointswitches are developed or if fewer input and output lines were required,a single crosspoint switch may be used to implement connectivity in theXPC.

FIG. 11 illustrates a data plane 44 of an example network element. Asshown in FIG. 11, IOCs 30 are connected via the connector plane to oneor more of the crosspoint switches 34, which switch the signals from theIOC 30 to one or more of the DSCs 32. The crosspoint switches 34 may allbe active and handling traffic on the network element or, alternatively,one of the crosspoint switches may be reserved and activated only uponfailure of one of the working crosspoint switches. In the illustratedembodiment there are two crosspoint switches. The invention is notlimited in this manner as more than two crosspoint switches may be usedas well. According to an embodiment of the invention, every IOC isconnected to at least one of the XPCs and all of the DSCs are likewiseconnected to that XPC to enable full interconnection between the IOCsand DSCs on the network element.

In the illustrated embodiment, the IOCs are connected to each crosspointswitch using three pairs of links. Since one of the XPCs is active forthe particular IOC, the links to the other XPC will remain inactiveuntil required. In the illustrated embodiment two of the links areconnected to a first DSC and a third of the links from the first IOC isconnected to a second IOC. Similarly, links from the second IOC areconnected to a second DSC. Any implementation may be possible and theinvention is not limited to this particular illustrated example. Forsimplicity not all of the connections implemented by the XPC have beenillustrated in FIG. 11.

Signals received by the DSC are processed and passed to the switchfabric 46 to be switched between interfaces on the network element. Theswitch fabric 46 may be a dynamic non-blocking switch fabricarchitecture. Switch fabrics are well known in the industry and anyconventional switch fabric may be used to switch packets between thedifferent interfaces on the network elements. On the reverse path fromthe switch fabric to the IOCs, the packets will take the reverse pathfirst traversing a DSC, then passing through one of the crosspointswitches, and then ultimately being formatted for transmission by one ormore of the IOCs.

An example of a network element configured to use a data plane of thisnature is illustrated in FIG. 12. As shown in FIG. 12, the networkelement includes a data plane 44 configured to handle data traffic onthe network and a control plane 48 configured to enable higher levelcontrol of the network element to take place. In the illustratedembodiment, data traffic is received at the IOCs 30 and transferredthrough links in the midplane 24 to one or more of the XPCs 34 whichcontrol the interconnection between IOCs 30 and DSCs 32. The DSCs 32receive the data traffic and perform packet processing on the receiveddata traffic.

In the example data plane 44 illustrated in FIG. 12, traffic is receivedat a crosspoint multiplexer 50 which operates to select one or moreactive links from the available links is passed to an ingress ASIC 52.The ingress ASIC is supported by an ingress network processor 54 thatperforms data path servicing operations on the data. Optionally, amemory 56 may be provided to store data and instructions for executionby the ingress network processor 54. The data is then prepared to beforwarded to a switch fabric interface 58.

Packets or other logical associations of data are then passed to theswitch fabric interface 58, switched in the switch fabric 46, andundergo additional processing on the reverse path through the DSC. Forexample, in the illustrated embodiment an egress ASIC 60 receives thepackets, and strips off whatever overhead was added to enable the datato traverse through the switch fabric. Optionally, additional postswitching processing may be performed on the data via egress ASIC 60 andassociated egress network processor 62. The processed data is thenpassed to egress crosspoint multiplexer 64 which controls selection oflinks to cause the data to be passed to the appropriate IOC via one ormore of the XPCs. After post switching processing in the DSC, thepackets are passed via the midplane to the crosspoint switch where theyare directed to the appropriate output IOC.

The control plane of the network element is configured to controloperation of the network element and provides an interface to theexternal world to allow the network element to be controlled by anetwork manager. In the illustrated embodiment, the control planeincludes a processor 66 executing control logic 68 that enables controloperations to be executed on the network element. For example, thecontrol logic 68 may include software subroutines and other programs toenable the network element to engage in signaling 70, routing 72, andother protocol exchanges 74 on the communication network. The inventionis not limited to any particular implementation of the control plane 48as numerous control planes may be used in connection with the dataplanearchitectures described herein.

According to an embodiment of the invention, the control logic isconfigured to implement a crosspoint control process 76 to enable thecrosspoint switch to be programmed to interconnect particular IOCs withother IOCs, to interconnect IOCs with particular DSCs, interconnectDSCs, and to otherwise control interconnection of functional cards onthe dataplane of the network element. As shown in FIG. 12, the XPCcontrol may communicate with the DSCs, the XPCs, and the IOCs to allowthese components to be instructed as to which links are to be used tocommunicate data and which traces on the connector plane are to beinterconnected. For example, as mentioned above in connection with APSswitching, multiple IOCs may be transmitting data streams to aparticular DSC. The crosspoint control process 76 may be used toinstruct the DSC as to which of the 12 available links are currentlyactive, which of the currently active links are being used to carrytraffic, and which links are logically bundled together. Similarconfiguration information may be provided via the crosspoint controlprocess to the XPC and IOC cards as well. These and other controlfunctions may be implemented via the crosspoint control process and theinvention is not limited to the particular listed control functions.

Control instructions may be passed between the control process on thecontrol plane of the network element and the functional cards that willimplement the control instructions using out of band signaling overdedicated control lines as illustrated in FIG. 12. These control linesallow the control plane to set up connections between IOCs and DSCs andnotify the components of failures and other events that may change themanner in which communications take place between the functional cards.

Alternatively, the control program may communicate with a subset of thefunctional cards and enable the functional cards to communicate witheach other using in-band signaling to effect control of the system in adistributed fashion. For example, the control subsystem may communicatewith the DSCs and cause the DSCs to control operation of the IOCs usinga proprietary or open source protocol. In this example, according to oneembodiment of the invention, the management of IOCs is handled by thecontrol processor resident on the DSCs. The IOCs connected to the DSCare then managed by its control processor. A proprietary protocolsupports transport of packets (ingress and egress directions) as well ascontrol messages. These control messages may be transported in-bandalong with data as described above and as illustrated in connection withFIG. 8.

One protocol that may be used to effect control of the IOCs by the DSCs,according to an embodiment of the invention, includes three types ofcontrol messages: command messages, reply messages, and event messages.Command messages are sent from the control processor on a DSC to itsdesignated IOC. Reply messages are sent form an IOC to the controlprocessor on its designated DSC. These messages are generated inresponse to the command messages. Event messages are sent from an IOC tothe control processor on its designated DSC and are generally generateddue to the occurrence of a local event on the IOC, such as an interruptor a timeout. Although a proprietary protocol has been described, otherprotocols may be able to be used to communicate between the IOCs andDSCs via the XPT.

The protocol may be used in a number of ways to enable the IOCs and DSCsto work together. For example, a DSC may instruct an IOC to ceasetransmitting data on a particular link and start transmitting data onanother link. The IOC may output a response to the DSC upon completionof the instruction. These protocol exchanges of host messages areimplemented on the data channel between the IOC and DSC to preventduplicative control and data paths from being required between thesecomponents.

FIG. 13 illustrates a block diagram of an embodiment of the invention inwhich IOCs are connected to DSCs via midplane connections under thecontrol of the XPT. It should be noted, in connection with this figure,that the midplane connections are actual physical serial connectionsformed on a midplane in the network element. Thus, the signals from theIOC pass through a first set of serial connections on the midplane tothe XPC, are switched at the XPC to other serial connections on themidplane, and pass through those second serial connections on the samemidplane to the intended DSC. The midplane connections were discussed ingreater detail above.

As shown in FIG. 13, the IOC includes an IOC crosspoint (XPT) interfaceblock. The IOC XPT interface is responsible for transport of various I/Obus protocols over 1 to m high-speed serial links. The IOC crosspointinterface block receives command messages, interfaces with processingcircuitry on the IOC to implement the command, and issues reply messagesto the DSC. The IOC XPT block 78 also generates the event messages uponoccurrence of an event on the IOC. IOC control messages received at theIOC XPT block 78 are extracted and consumed locally on the IOC.

The DSC includes a DSC XPT interface block 80 which is responsible forthe transport of packets and control messages over 1 to n high-speedserial links. It generates messages for transportation to the IOCs andreceives reply and event messages from the IOCs.

The XPC is controlled by software, such as XPT control software, toprovide proper interconnection between the IOCs and DSCs. As discussedabove, the XPC includes an XPT I/F 42 to allow it to receiveconfiguration input from the control plane 48.

FIGS. 14-16 illustrate several protection schemes that may beimplemented using the front end described herein. As shown in FIG. 14,1:1 and 1:n sparing of IOCs is possible using the crosspoint switch 34to direct traffic between a given DSC 32 and alternative IOCs 30. Asdiscussed above, this may allow APS switching to occur at the DSC, forexample via an APS MUX 80 rather than at the IOC 30, to allow for IOCsparing in a SONET system. Where the DSC is not required to operate in aSONET environment, this function may be disabled. In FIG. 14 each of theIOCs 30 is illustrated as being configured to implement four OC-12interfaces. The invention is not limited in this manner as the IOCs mayimplement any desired number of interfaces at any desired line rate.

The IOCs are connected to the XPC via mid-plane links 84 and switched bythe XPC to other mid-plane links 86 to arrive at a desired DSC 32. TheDSC has an XPC multiplexer configured to selectively cause traffic to beactive on one of the spared IOCs. In the illustrated example the top IOCin FIG. 14 a has been designated as the active IOC and the bottom IOChas been designated as a spare IOC. FIGS. 14 b and 14 c illustratesimilar systems except that FIG. 14 b illustrates 1:n sparing and FIG.14 c illustrates m:n sparing.

FIG. 15 illustrates an embodiment of the invention in which XPCs arespared as well as IOCs are spared. Sparing of the XPCs allows an XPC tobe replaced upon occurrence of a failure in the XPC to thereby increasethe reliability of the network element. Since each XPC is non-blockingand provides full mesh connectivity between all inputs and all outputs,each XPC is capable of handling communication between the IOCs and DSCs.Thus, according to one embodiment, one or a given subset of the XPCs mayhandle all of the connectivity between IOCs and DSCs while allowing thespare XPC to remain idle. In an alternative embodiment, the spare XPCmay be configured to handle traffic while none of the XPCs isexperiencing failure, and the load may be redistributed to thenon-failing XPC(s) upon failure of one of the XPCs.

FIG. 16 illustrates an embodiment of the invention in which DSCs, XPCs,and IOCs are all spared. This allows DSCs to be replaced as failuresoccur on the DSCs. As shown in FIG. 16, the DSC XPT Mud allowsparticular XPCs to be selected to be used in transmitting signalsbetween the IOCs and DSCs. Selection of DSC may occur by programming theXPT to transfer signals from a given IOC to multiple DSCs andcontrolling the DSCs to cause one to operate as the default DSC and theother to operate as a spare DSC such that that DSC will handle signalsonly upon failure of the default DSC. Alternatively, the XPT may beconfigured to transfer signals from selected IOCs to a given DSC and totransfer the signals from the selected IOCs to another given DSC or agroup of other DSCs upon notification of a failure of the primary DSC.Other methods of sparing DSCs may be possible as well and the inventionis not limited by the actual manner in which a change in control betweenspared DSCs is effected.

For example, FIG. 17 illustrates several various combinations of sparingthat may be implemented in a network element. As shown in FIG. 17,sparing of IOCs is independent of the manner in which DSCs are spared,so that multiple combinations of sparing scenarios may occur.Specifically, as shown in FIG. 17, 1:1, 1:n, and m:n sparing may occurat the IOC side, while using 1:1 sparing on the DSC side. Similarly,1:1, 1:n, and m:n sparing may occur at the IOC side, while using 1:nsparing on the DSC side or while using m:n sparing on the DSC side.Sparing of the IOCs and DSCs is thus not interrelated as any desiredsparing implementation may occur. Sparing of the IOCs and DSCs is alsoindependent of any desired sparing of the XPCs as discussed in greaterdetail above.

The control plane programs may be implemented in computer software andhosted by one or more the CPUs on the network element. Alternatively,the control plane may be implemented external to the network element andcontrol information may be communicated to the data plane via acommunication system such as a network management system connected to adedicated management port.

The functions described above may be implemented as a set of programinstructions that are stored in a computer readable memory within thenetwork element and executed on one or more processors within thenetwork element. However, it will be apparent to a skilled artisan thatall logic described herein can be embodied using discrete components,integrated circuitry such as an Application Specific Integrated Circuit(ASIC), programmable logic used in conjunction with a programmable logicdevice such as a Field Programmable Gate Array (FPGA) or microprocessor,a state machine, or any other device including any combination thereof.Programmable logic can be fixed temporarily or permanently in a tangiblemedium such as a read-only memory chip, a computer memory, a disk, orother storage medium. Programmable logic can also be fixed in a computerdata signal embodied in a carrier wave, allowing the programmable logicto be transmitted over an interface such as a computer bus orcommunication network. All such embodiments are intended to fall withinthe scope of the present invention.

It should be understood that all functional statements made hereindescribing the functions to be performed by the methods of the inventionmay be performed by software programs implemented utilizing subroutinesand other programming techniques known to those of ordinary skill in theart. Alternatively, these functions may be implemented in hardware,firmware, or a combination of hardware, software, and firmware. Theinvention is thus not limited to a particular implementation.

It should be understood that various changes and modifications of theembodiments shown in the drawings and described in the specification maybe made within the spirit and scope of the present invention.Accordingly, it is intended that all matter contained in the abovedescription and shown in the accompanying drawings be interpreted in anillustrative and not in a limiting sense. The invention is limited onlyas defined in the following claims and the equivalents thereto.

1. A network element, comprising: a plurality of Input/Output Cards(IOCs); a plurality of Datapath Service Cards (DSCs); and at least afirst crosspoint switch card (XPC) configured to be able to selectivelyinterconnect each of said IOCs with each of said DSCs.
 2. The networkelement of claim 1, further comprising a switch fabric configured todynamically interconnect outputs of said DSCs.
 3. The network element ofclaim 1, further comprising a second XPC configured to selectivelyinterconnect each of said IOCs with each of said DSCs, said second XPCforming a spare XPC for said first XPC.
 4. The network element of claim1, wherein a first subset of the IOCs are working IOCs, a second subsetof the IOCs are spare IOCs.
 5. The network element of claim 4, whereinthe spare IOCs are configured to spare the working IOCs in an m:nfashion.
 6. The network element of claim 1, wherein a first subset ofthe DSCs are working DSCs, and a second subset of the DSCs are spareDSCs.
 7. The network element of claim 6, wherein the spare DSCs areconfigured to spare the working DSCs in an m:n fashion.
 8. The networkelement of claim 1, wherein the XPC is further configured to be able toselectively interconnect IOCs with other IOCs
 9. The network element ofclaim 1, wherein the XPC is further configured to be able to selectivelyinterconnect DSCs with other DSCs.
 10. The network element of claim 1,wherein the network element is a SONET switch, and wherein AutomaticProtection Switching (APS) is performed at the DSC.
 11. The networkelement of claim 10, wherein a first fiber used on a SONET ring is homedat a first IOC, wherein a second fiber used on the SONET ring is homedat a second IOC, and wherein the DSC enables APS to be performed betweenthe first and second IOCs.
 12. The network element of claim 1, furthercomprising a second XPC, said first XPC having a first set of inputs anda first set of outputs, and said second XPC having a second set ofinputs and a second set of outputs, and wherein a plurality of saidfirst outputs are connected to a plurality of said second inputs andwherein a plurality of said second outputs are connected to a pluralityof said first inputs.
 13. A network architecture for a network element,comprising: a control plane; and a data plane, said data plane having aplurality of functional cards and a crosspoint switching structurehaving an ability to selectively interconnect any of said functionalcards with any other of said functional cards.
 14. The networkarchitecture of claim 13, wherein the crosspoint switching structurecomprises redundant crosspoint switches.
 15. The network architecture ofclaim 13, wherein the data plane comprises a midplane having a pluralityof traces extending between the functional cards and the crosspointswitching structure, but does not have direct traces providing directmesh interconnectivity between the functional cards.
 16. A method ofimplementing Automatic Protection Switching in a network element havingInput/Output Cards (IOCs) and Data Service Cards (DSCs) connected in aany-to-any fashion, the method comprising the steps of: receivingstreams of SONET traffic at a plurality of independent andnon-interconnected IOCs; and selecting one of the streams of traffic forprocessing at a DSC affiliated with said plurality of IOCs.